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ABSTRACT 

Change process research had developed some practical 
tools and concepts which can assist in the determination of essential 
variables for an evaluation design. The Concerns-Based Adoption Model 
(CBAM) Project at the Texas Research and Development Center in 
particular has developed specific measures which can be applied 
directly: "Levels of Use of the Innovation" measure evaluates to what 
extent a program is actually being used^ while "Innovation 
Configuration" describes what adaptations, if any, have been made in 
the program. The need to view any program within a time frame ^ 
necessary for it to be considered fully installed has^ also been a 
valuable contribution from the change process research. The two CBAM 
measures can also be used to establish what this time period is. 
Taking into account the three dimensions of time for implementation, 
levels of use, and description of use, evaluation designs can be far 
more specific about what they portend to evaluate and can attribute 
effects of this specifically described program with a greater degree 
of validity. (Author/PN) 
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Abstract 

Bilingual education is gaining more attention'as many new programs are 
being tested. Policy decisions on whether to continue, alter or terminate 
bilingual programs are often based on evaluation studies of them. Ttie eval- 
uation of a program as it is supposed to be^is one thing, but the evaluation 
of the pi^ogram which is actually being used'^is another matter entirely. 
Programs are very often changed by the user, sometimes so much that they are 
not recognizable as the original model. Evaluation designs must first of all 
describe exactly what program is being studied before its effects can be 
accurately correlated. Furthermore, evaluation designs must determine* if a 
program, whether "pure" or "adapted," is actually being used in order to know 
if, indeed, there are any effects to be assessed. Unfortunately, these 
determinations are often not made before an evaluation is conducted, with 
results thus being susceptible in invalid assessment and policy decisions 
being made on the basis of misleading information.. 

Change process research has developed some practical tools and concepts 
which can assist in the determination of these essential variables for an 
evaluation design. The Concerns-Based Adoption Model Project at the Texas 
Research and Development Center in particular has developed specific measures 
which can be applied directly: Levels of Use of the Innovation measures to 
what extent a program is actually being used, while Innovation Configuration 
describes what adaptations, if any, have been made in the program. The need 
to view any program within a time frame. necessary for it to be considered 
fully installed has also been a valuable contribution from the change process 
research. The two CBAM measures can also be used to establish what this time 
period^is. Taking into account the three dimensions of time for implementation, 
levels of use, and description of use, evaluation designs can be far more 
specific about what they portend to evaluate and can attribute effects of this 
specifically described pragram with a greater degree of validity. 
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EVALUATION OF THE DELIVERY OF SERVICES: ^ 
A CONCERNS-BASED PERSPECTIVE FOR THE DESIGN OF EVALUATIONS^ 

Gene E, Hall 

Research on the Improvement of Practice in Schools and Colleges 
Research and Development Center for Teacher Education 
The University of Texas at Austin ^ 

Program development has significantly increased in sophistication and 
effectiveness during the last fifteen years. Program development in bilingual 
education clearly reflects this general trend. There is new sophistication Jn 
terms of concepts, theories, and models for bilingual programs and the different 
operational programs around the country provide a rich diversity of applications 
of these concepts and theories to practice. During this period there has also 
been increasing sophistication in theories, models, and approaches to evalua- 
tion. However, evaluation as an' applied science still is facing a great chal- 
lenge in being able to handle effectively the complexities of these sophisti- 
cated programs, 

- ^ There are some recent conceptual and methodological breakthroughs that can 
significantly contribute to the power of evaluation studies, some of which are 
from disciplines other than evaluation itself. The interdisciplinary nature of 
evaluation makes possible tire integration of knowledge from other fieljjs, and it 
is to their credit that bilingual program evaluators have been willing to incor- 
porate approaches and findings from other areas. Research on the change pro- 
cess and on implementation represents one area which has been able to make valu- 
able contributions to evaluation processes. This paper briefly explores some 
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of the developments from change research which can be of help to evaluators. 
Two major concepts are focysed upon, that of Levels of Use of the Innovation and 
that of Innovation Configuration . Illustrations and sample data "from the 
field" will be used to explain these concepts and their relative measurement 
techniques. The paper also explores some implications of these perspectives on 
change to th'e evaluation of bilingual programs. 

Perspective 

Recent research on change places emphasis on viewing change as a process , 
not an event. Practitioners, researchers, evaluators, atid policy makers are 
showing increasing sensitivity and awareness to this assumption. Three' key 
parameters of the change process research that are important to consider in the 
design of an evaluation study are: 

1. - implementation is a phase of the change process; 

2. innovations are usually adapted during implementation; and 

3. attention musf^be given to individual users and nonusers 
of an innovation. 

Addressing these parameters has been crucial to understanding, managing, and 
assessing the effects of change efforts. 

The j§election of designs and measures' for education studies should also 
reflect awareness of these parameters. Both summative, or product, and forma- 
tive, or process evaluation have valuable roles to contribute, but the appropri- 
ate approach must be used at the appropriate time. Exploring these three vari- 
ables can help define whether summat'ive or formative designs are' to be used, and 
which variables should be e^mphasized. 

Change occurs over time. In a key synthesis of the 'literature, Fullan and 
Pcmfret (1975) articulate the distinctions between development, dissemination. 



the decision to adopt, and implementat lojq. As they point out, implementation is 
the phase of the change process that occurs, after the program is introduced into 
the classroom, building or school district, they emphasize, however, that full 
use of the program is not" instantaneous and is no\ achieved without problems. 
Formative evaluation is particularly relevant during this phase. Program imple- 
mentors need information about how the change effort is progressing. This feed- 
back is essentia^l if the implementation of a program is to twore quickly and 
easily become a routine pattern of uses. Contrary to much evaluation practice,^ 
the implementation phase is a poor time to be conducting summative evaluations. 
, The term innovation refers to the new program or process being implemented 
or to key changes being made if) an existing program. Recognition that the 
innovation can, and probably will, be adapted by different users to fit dif- 
ferent contexts is another major finding of the change process research that has 
implications for evaluation. Perhaps the most well-known reference on adapta- 
tion of the innovation is thet Rand change agent study (Berman & McLaughlin, 
1975) in which the authors propose the concept of "mutual adaptation." The 
innovation is adapted to fit the local context and the local users rather than 
being implemented exactly as prescribed by its developers. 

At the same time, the users adapt somewhat to fit the requirements of the inno- 
vation. Thus, both the innovation and the users adapt to more closely approxi- 
mate the requirements of the other. The implication of this concept for evalua- 
tion is that evaluators cannot assume that the innovation as implemented at each 
site is a close formulation of the innovation as espoused by the program devel- 
oper, or that the innovation is implemented in the same way in each site. 

Individuals involved in change are a key focus of research based on the 
Concerns-Based Adoption Model focuses on (Hall, Wallace & Dossett, 1973). The 
CBAM model recognizes implementation and innovation adaptation, and then brings 



,an additional emphasis to the understanding of change as a process by focusing 
on the individual. From the CBAM perspective each individual will adop.t the 
innovation in their own way, and in that respect each individually is .qpnsidered 
to be either a user or a nonuser. CBAM experience suggests, that in general, 
administrators and other decision-makers are not reliable sources oY information 
as to what actual classroom practice is for each individual. Therefore,, evalua- 
tors should be encouraged to assess directly whether each individual is a user 
or nonuser. \ 

In summary, recent change research focuses on the Individual , acknowledge- 
ment of the adaptation of innovations, and the recognition of the implementation 
phase of the change process. Each of these parameters has important implica- 
tionsWor the design of program evaluations. Evaluation studies cannot consider 
a program as an event which automatically assumes full functioning simply by 
being placed in the classroom. Instead, use or nonuse of the innovation must be 
directly assessed for each teacher. The operational form of the innovation must 

'also be assessed individually. Furthermore, evaluation studies must take into 
account change as a process, thereby utilizing both formative and summalive 
evaluations and conducting studies over time, rather than just gathering data 
at one or two points in time and issuing a single report. 

Two Basic Questions 

The literature on change emphasizes the importance of understanding what 
happens in individual classrooms through direct assessment. It is not possible 
to make interpretations about how the process of change and implementation of 
new programs is unfolding or to draw conclusions about tho^ effects of new pro- 
grams, without knowing first-hand what is happening in each classroom. There 
are two basic questions that must be -^sked before an evaluation study can pro- 
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ceed, for either pr^ocess or product evaluations. The simplest way to ask these 
questions regarding the program being examined is: 

/For each teacher, school and district, • • ' 

1. IS IT BEING USED? . ^ 

' 2. -WHAT IS IT? 

" The first question asks simply whether or not each teacher who is supposed 

•5. 
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to be using the innovation in fact is. Incidently, the first question must also 
be asked of any compariso^ groups being used in the study. The second question 
addresses the issue of innovation adaptation. This question ^sks exactly what 
operational form of the innovation 'is being used by each "user." In the Con- . 
cerns-Based Adoption Model (CBAM) research, direct efforts have ^been made to. 
determine use or nonuse of the innovation through a concept called Levels of Use 
0?' the Innovation .'^' Also in the' C|AM, ' research attempts have been made to des- 
cribe the various operational forms of the innovation which could be implemented 
in different classrooms. The concept of Innovation Configurations has befen 
developed to describe the phenorhenon. 

Is It Being Used? r 

Regardless of whether a proc*ess or product evaluation is being'^done, "is 
the program being used?" i^^the first question th-at evaluators need to ask with 
reg'ard to any program. Although this question seems obvious and straight for- 
ward, it is alarming to review evaluation and research studies in which it has 
•been neither directly nor indirectly asked or answered. The concept of Levels 
of Use has proven to be very useful "in answering this question. 

Levels of Use of the Innovation (Hall, Loucks, Rutherford & Newlove, 1975) 
describes what individuals and groups are doing or not doing with an innovation. 
The concept focuses on whether or not and how they afe using the innovation. 
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The first major breakout that levels of Use (LoU) reveals i^to distinguish 
operationally between users and nonusers. A further breakout is possible into 
i ej^^t^. distinct Levels of Use, including three levels of nonuse. The eight 
Leye$s of Use 3re surTimari2,ed^in Figure 1. 

More specifically, the nonuse levels range from the person at a Level of 



,0 

Use 0, Nanuse, is^not doing anything relative to the program, ;to a. Level of 
=^IJse r*oVi^e^^ti^ is considering or exploring use of the program but 

has not (lecided to use it, to a Level of Use II Preparation person who has macie 
a decisis to use the^rogram and is preparing for first use. For each of ttrese 
nonuse levels, ybehfvioral indicators have been identified and a set of cate- 
gories can be useoTo further describe" the actual behaviors. \ 

The five use levels arenas follows: Ley^ of Use III person^ are at a 
Mephanical Leve^l of Use-^r&this tends to be a time that is typically observed in 
early users-of .an inno|vatr6n. Use is disjointed and the person has a very 
•short-term focus in relation to their use of the program. Later on v^e find that 
persons move^on to being at a Level of Use IVA Routine. These persons have an 
established pattern of use and are makiftg-few, if any, adaptations' in their use 
of the pr'^ogram. Some individuals move on to a Level of Use IVB Refinement where 
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they are making adaptations- which angl' for student benefit. Level of Use .V Inte-^ 
gration people are making adaptations in use of the innovation by collaborating 
with other users of the innovation so that collectively they can have greater 
impact. The rare Level of Use VI Renewal person is one who is making major mod- 
ifications in the program or considering an alternative to the present innova- 
tion. 

These eight Levels of Use have been" defined in the Level of Use chart and a 
measurement procedure has been developed. Th4^ea$urement procedure which has 
been demonstrated to be reliable and valid, uses a foi;used Interview (Loucks, 



Figure 1 i 



LEVELS OF USc OF THE iriflOVATION 



LEVELS OF USE 



DEFINITION OF USE 



0 NONUSE 



Decision Point A 



STATE IN WHICH THE USER HAS LITTLE OR HO K.NOV/LEDfiE CF 
THE INNOVATION, NO INVOLVEMENT WITH THE INNOVATIOfU 
AND IS DOING NOTHING TOWARD BECOf'lING INVOLVED. 

Takes action to learn more detailed information about the 
innovation. 



I ORIEilTATION state in which the user has recently acouired or is 

ACQUIRING information ABOUT THE INNOVATION ANO/OR 
HAS RECENTLY EXPLORED OR IS EXPLORING ITS VALUE 
ORIENTATION AND ITS DEMANDS UPON USER AND USER 
SYSTEM. 

Decision Point B Makes a decision to use the innovation by establishing a 
time to begin. 

II PREPARATIOfI STATE in which the user is preparing FOR FIRST USE 

OF THE INNOVATION. 

Decision Point C Changes, if any, and use are dominated by user needs. 

Ill HECHAinCAL USE state in which the user focuses most z^fort on the 

SHORT-TERM, DAY-TO-DAY USE OF THE INNCVATION WITH 
LITTLE TIME FOR REFLECTION. CHANGES IN USE ARE 
MADE MORE TO MEET USER NEEDS THAN CLIENT ;,'EEDS . 
THE USE^. IS PRIMARILY ENGAGED IN A STE^'WISE ATTEMPT 
TO MASTER THE TASKS REQUIRED TO USE THE INNCVATION, 
OFTEN RESULTING IN DISJOINTED AND SUPERFICIAL USE. 

Decision Point D-1 A rountine pattern' of use is established. 

IVA ROUTINE USE of the innovation is stabilized, few, if any, 

CHANGES ARE BEING MADE IN ONGOING USE. LITTLE PREP- 
ARATION OR THOUbnT IS BEING GIVEN TO IMPROVING INuG- 
VATION USE OR ITS CONSEQUENCES. 

Decision Point D-2 Changes use of the innovation based on formal or informal 
evaluation in order to increase client outcomes. 



IV3 REFINEMENT 



Decision Point E 



Decision Point F 



VI RENEWAL 



STATE IN WHICH THE USER VARIES THE USE OF THE INNO- 
VATION TO INCREASE THE IMPACT ON CLIENTS WITHIN THE 
IMMEDIATE SPHERE OF INFLUENCE. VARIATIONS ARE BASED 
ON KNOWLEDGE OF BOTH SHORT- AND LONG-TE.RM CONSEQUENCES 
FOR CLIENTS. 

Initiates changes in use of innovation based on inout of and 
in coordination with what colleagues are doing, 

INTEGRATION stat£ in which the user is combining own efforts to 

USE THE INNOVATION' WITH RELATED ACTIVITIES OF COL- 
LEAGUES ^TO ACHIEVE A COLLECTIVE IMPACT ON CLIENTS 
WITHIN THEIR COMMON- SPHERE OF INFLUENCE. 

Begins exploring alternatives to or major modifications of 
the innovation presently in use. 

STATE IN WHICH THE USER REEVALUATES THE QUALITY OF 
USE OF THE INNOVATION, SEEKS MAJOR MOD I I CAT IONS OF 
OR ALTERNATIVES TO PRESENT INNOVATION TO ACHIEVE " 
INCREASED IMPACT ON CLIENTS, EXAMINES NEW DEVELOP- 
MENTS IN THE FIELD, AND EXPLORES NEW GOALS FOR SELF 
AND 'THE SYSTEM. 



rrcm:^ The IzU Chart. Austin: Research and Development Center for Teacher 
« Education, The University of Texas, 1975. 
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Newlove & Hall, 1976). The interview is generic in terms of its process and 
therefore can be applied to different programs and processes. The interview 
takes the form of a general conversation with a teacher who is asked to describe 
behavioral ly what they are doing or not doing with the innovation. The inter- 
view is constructed using a branching format based on the operational defini- 
tions of Levels of Use. Training manuals and systems hdve been developed for 
training evaluation staff to certifiable levels in terms of their proficiency in 
conducting Levels of Use interviews and rating interview data. 

There are several interesting implications for evaluation that can be de- 
rived from the Levels of Use research. For example, the proportion of individ- 
uals that are found at each Level of Use is not^equal (See Table 1). In a 
stratified sample of innovation users and nonusers based upon amount of experi- 
ence with an innovation, ft is commonly found that the largest sample of users * 
are at LoU IVA Routine. Lou V and VI individuals are relatively rare in most 
stratified samples. Another pattern that is typically observed is that 60 to 
70% of the first-year users of an innovation will be at LoU III Mechanical Use 
(Hall & Loucks, 1977). 

One implication of this last finding is that it can be assumed with some- 
thing as complicated as a bilingual program that many, if not all, firs£ year 
users of the program will be at a Mechanical Level of Use. This is a time for 
conducting process or formative evaluations and not a time for conducting a sum- 
mative evaluation. Persons at a Mechanical Level of Use may be doing a more 
awkward and disjointed job with the new approach than they were previously with 
their past practice. It is not a fair time for uhe individual br the new pro- - 
gram to be assessed in terms of outcomes and ultimate effects. 

A useful application of the Levels of Use concept, especially for formative 
or process evaluation, is to provide a benchmark for the rate and degree of 



implementation which is occurring;' A study using the LoU technique to monitor a 
three-year effort ' involving a revised science curriculum provides an example. A 
summary of this study's data is presented in Table 2, ; evels of Use were 
assessed five times during the three-year period, while a concerns-based staff 
development effort was also underway (Loucks & Pratt, 1979), As part of the 
staff development plan, the LoU data were immediately fed back to the staff 
developers so that staff development experiences could be adapted according to 
the Levels of Use of teachers in various parts of the school district* The LoU 
data were also used :o illustrate to district policy makers that implementation 
V^as not an event and th-at use was moving ahead. 

Looking at the data in Table ?, several treads can be readily discerned. 
First of all, the number of individuals, at LoU 0 Nonuse rapidly decreases as 
the implementation effort progresses in time. There is also a definite trend 
towards use (LoU. Ill - VI) of the revised cur^iclum materials. One of the 
interesting findings relative to the process of implementation is that there was 
still a large proportio^n of individuals at a Mechanical Level of Use after three 
years. This led the staff developers in this particular situation to acknowl- 
edge that implementation of any new curriculum takes longer than is typically 
assumed; and, furthermore, staff development to assist teachers in moving to 
higher Levels of Use requires a different plan than does a staff development 

if 

effort to assist teachers in moving from nonuse to an initial stable pattern of 
use. 

The LoU concept has been applied in a bilingual education program setting 
by Domipguez, Tumner and Jackson (1980). In their study of the implementation 
of bilingual education programs in urban and rural sites, they too found a dis- 
tribution of Levels of Use ranging from LoU 0 Nonuse to LoU IVB Refinement. 
Because of the complexity of bilingual education being actually a ''bundle" of 
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innovations rather than a single one, they assessed' Levels of Use for various 
innovations within the bund'le. For example, they distinguished .between ESL, 
Spanish reading, and ESL Spanish math. Thus, they were able to provide more 
precision in distinguishing between users and nonusers of various components in 
tjieir studies' instead of simply referring, to users and nonusers of a composite 
bilingual education program,* 

Another important application of the Levels of Use concept is to insure 
that^a comparison or control group, does not have users of the materials within 
it. In other words. Levels of Use must also be assessed for the group that ^^is 
assumed to be nonusers. In one study (rfall & Loucks, 1977), an evaluation was 
done of 'Individual ly Guided Education, an inYiovation bundle involving individ- 
ualized instruction, multi-aged teaming and several other major innovations. 
This study observed that in the treatment schools in which the innovat^ion of 
individualized instruction had supposedly been imp.lemented for three years, .only 
80% of the^teachers were In fact using individualized instruction. Use/nonuse 
data were then analyzed for the comparison schools, whj^ch were matched with 
elever? treatment schools. It was observed that 63% of the. teachers in the com-, 
parison schools were in fact individualizing their instruction in reading! 
Both comparison and treatment schools had large proportions of users of a key 
component of the innovation. • ^ 

• In this type of evaluation study, it would be impossible to draw conclu-^ 
sions about the effects of the^innovati ve program when in fact key specific 
innovations within the 'innovation bundle were in use in both expef^imental and 
control schools. Unfortunately, what too frequently happens in these types of 
evaVQation designs Is that rather than assessing use or ponuse, the inference is 
drawn based upon the statements of principals or district administrators that in 
fact one school is a treatment school and the other school a nonuser school. 



When the achievement t^^sts or other outcome data are collected and analyzed the 
conclusion is made that there are "no significant differences". Then a policy 
decision immediately follows that the new way is no better than the old and the 
new way is eliminated. When in fact, the new way.^nd the old way have not 
rec^^ived a fair test. 

That too much is taken for granted in assessing use or nonuse seems to be 
an ongoing problem in the coYiduct of innovation studies. Moreover, th^ere are 
too many evaluation studies that do not acknowledge the existence of the imple- 
mentation phase. These studies continue to be based on the assumption that 
change is an event without the transition period involving Mechanical Level of 
Use. These kinds of studies further assume that once teachers have received 
program materials and a one or two day workshop in August, they then automati- 
cally become sophisticated users and are capable of functioning proficiently in 
the programs as of September. Although this may sound like a crazy approach to 
evaluation, it still goes on. 

What Is "It"? . 

This basic question has also received attention in the change research. 
Obviously, in many instances the innovation is adapted during the adoption pro- 
ces<?^TheSe adaptations are often appropriate and well within the scope of the 
original intent of the innovation as outlined by its developers. In other 
instances theses adaptations in the innovation may be beypndassumedf limits of 
the program. With some of these adaptations the program could conceivably have 
different effec^^S than were originally intended. In CBAM, research of this phe 
nomenon has been addressed through the concept of Innovation Configurations 
(Hall & Loucks, 1978). Innovation Configurations as a tool provides a means to 
describe what happens to the innovation as it is adopted by different users. 
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Again, the initial focus is upon the individual user, which 'can then be aggre- 
gated for the school or school district levels. 

The simplest question to be asked here is, ''what are the program^ users 
using?i' The program will presumably contain distinctive features, which often ^ 
include the use of certain materials and special roles which the students and 
teachers are to carry out. How does an evaluator determine the actual opera- 
tional form of the innovation that is in place? Conceptually, the problem is^" 
fairly easily understood, but is a fairly complex problem methodologically. An 
^An innovation can be considered to consist of a series of major components 
of which users of the program can be assessed for thier use or nonuse. For 
example, a continuous-progress mathematics curriculum could consist of: 1) the 
components of instructional materials used, 2) the grotiping of students, 3) and 
testing. Then in terms of use/nonuse, the*simplest llecision would be to judge 
as users "Df the program those teachers who were: 1) using the materials, 2) 
grouping students, and 3) testing them. However, the way^ that the materials are 
used, how students are grouped, and exactly how and what is done^with test 
results can be important variations . Thus, in the concept of Innovation Config- 
urations, the components are further divided into different "variations" of use 
which can be Identified. This example can briefly illustrate how variations are 
identified within each component: 

K Instructional Materials Component 

. a; Program materials only" 

b. Program materials plus basic text 

c. Text only 

d . Teacher made 

2. Grouping Component 

a. Large heterogeneous group 

b. Large homogeneous group 

c. Sn>al 1 groups 

d. Completely individualized 

16 
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3. Testing Component 1 

a. Testing done once eyery six weeks with nothing done with testing 
results 

b. Testing done weekly with test results fed back to students 

c. Students test themselves as they complete each objective 

In conducting an evaluation study, it would be the job of the evaluators to 
develop and conjplete an infiovation- configuration checklist '(See Figure 2) that 
consisted of these components and their variations. For each subject, ^an inno- 
vation configuration checklist would be cortipleted. A certain combination of 
responses across the components such as: (1-a) uses program materials only, 
(2-c) small groups of students, and (3-a) testing once every six weeks; would ' 
represent one "configuration" of the program. Any other combination of compo- 
nents would represent a different configuration. The evaluator could then sum- 
marize which configurations are in use in different clas.srooms. The concept of 
Innovation Configurations is thus used to define and describe the operational 
forms of the innovation. This entails describing: 1) what students and 
teachers are doing, 2) how materials and processes are used, and 3) what would, 
be observed when the program is in actual use. 

It is important to note that the concep^-n)f^ Innovation Configurations does 
not, however, describe implementation requirements. An implementation require- 
ment would be posse5sion-€f the textbooks. How the textbooks are used would be 
an Innovation Configuration component. This same rationale applies to outcomes. 
Outcomes and .effects of a particular innovation are outside of the innovation 
configuration itself. They are the result of using a particular configuration. 

These distinctions between implementation requirements, innovation config- , 
urations and outcomes become particularly important for the evaluator. An • 
evaluator needs to be able to determine what effects are a result of use of the 
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Figure 2 15 
Revised innovation configuration checklist for math curriculum study. 



I.D. (Last 4 digits of SS//) 



Please check one choice under each of the five categories below that is the most descriptive 
of your math instruction. 

1. Instructional* resources used: 

Program ^materials (i.e., packets, worksheets) only 

• Textbook(s) only 

Textbook(s) emphasized, program materials supplemental 

[Program materials emphasl,zed, textbook(sX supplemental 

Combination of text(s) and/or program materials with teacher-made materials and 

games 

Large variety of text(s), program materials, games, teacher-made materials, 
manipulatives , centers, labs, etc. 

2. Grouping patterns: 

Teach whole class or two groups in a class with children ranging in abilities 

^i.e., heterogeneous) " , 
Teach whole class or two groups in a class v/ith children of generally the 

same abilities (i.e., homogeneous) ^ ^ 
Teach 3 or more small groups that are fairly stable — the children in each^ 

group seldom* move to a different group 
^ Teach 3 or more small'^groups that change continually — the children frequently 

move to a different ^r'oup. 

Teach individuals only, no grouping 



3. Clusters of objectives; . ^ 

Use program clusters, largely in sequence as a framework for instruction 

Use program clusters largely ou,t of sequence 

Do not use program clusters 

4. Objectives: 

Use program objectives largely in sequence within the clusters 

Use objectives largely out of sequence 

^ Do not use program objectives 

5. Kind of testing: 

Use posttest on objectives (either program-supplied or teacher-made) and cluster 

test (either program-supplied or teacher-made) 

Use cluster test (either prograiB-supplied or teacher-made) only 

Use posttest on objectives (Either program-supplied or teacher-made) only 

Use teacher judgement only 

Varies widely using posttests, cluster tests and/or teacher judgement 

6. The use of test results: " . 

Each individual child is-*assigned work or activity depending on the results of 

the test given 

Wlien most of the group passes a test the group goes on; those who fail are 

given special attention 
Wiien most of the group passes a test the group goes on; those who fail will have 

anotlier chance to* learn later due to the spiral nature of the curriculum 
What is done with test results depends on the objectives being taught 



riRE THERE OTHER SIGNIFICANT FEATURES OF YOUR MATH PROGRAM THAT WE HAVE NOT INCLUDED? 



t^^;picase doacribe on b^ick of this pane) 



innovation. In order to do so it must be determined first that the innovation 
is being used and second wliat configuration of the innovation is being used. As 
previously pointed ouju, it is conceivable that different configurations of the 
innovation will result in different outcomes. The process evaluator may want to 
present pgrely -descriptive information about the different configurations that , 
are in use so that th^'e staff developers can plan specialized inservice which 

addresses the needs of teachers who are using different configurations. It is 

- J * 

also 'conceivable that through early testing certain desired learning pucomes 
may be associated with Certain configurations. These data could then been fed 
back so that staff development again could be targeted -to facilitate implementa- 
tion of tho^se configurations whigii^ are associated with more of the desired out- 
comes. Hall & Loucks (1980); Pratt, Winters' & George (1980) and George & Hord 
(1980), are papers that explore some of these^impl ications further. . 

The 'Innovation Configuration framework, has been applied in at least one 
study of bilingual^ education. Dominguez, Tumner and Jackson, (1980) identified 
a range of innovation configurations for bilingual education programs. Because 
of the elaborate nature of bilingual education, it was necessary to describe not 
only the configurations of bil'ingual education in terms of classroom practice, 
but also the school building and district level aspects of the configurations. 
Furthermore, they took advantage of the concept of an innovation configuration 
continuum (Hall & Loucks, 1978). 

The Innovation Configuration continuum (See Figure 3) suggests that the 
various configurations of a program can be placed along a continuum that ranges 
from those that most closely approximate the developer's model, to those, that 
are somewhat related to it and finally to those that are clearly deviant from 
the model as described by the developer. Using this procedure, Dominguez, et 
al., were able to distinguish between programs that were approximating the 



Figure 3 
Configuration Continuum 
Using **Car** as the ^Innovation 



Not the innovation 







UP CFP DP 



The innovation 
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Model (s) 



Points of Drastic Mucatlon 

UP — User's Point 

CFP — Change Facilitator's Point 

DP — • Developer's Point 
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ideals of bilingual education, (maint'enance), those that were on their way 
towards it (transitional), and those that w.ere clearly not within acceptable 
limits with regard to bilingual education theories and models (non-bilingual), 
(Oominguez et al . , 1980). Many of the participants within the settings of the 
non-bilingual programs may have claimed, and even believed, they were using 
b'il ingual ^education materials. However, in terms of the criterion that were set 
up to describe ideal models of bilingual education, their conf ig^irations were 
not in the ballpark. 

For the process evaluator, feedback about which configurations are in use 
and which components of innovations seem to be most clearly associated with de- 
sired outcomes can be of great value. The staff developers can fine tune and 
focus their staff development activities to address teacher needs in relation to 
certain 'components. Policy makers and administrators can understand better 
which resources and support to deliver so that the configurations put into use 
are those that are des-^red And all parties can be more concrete in their com- 
munications with, and understanding of each other. 

In Summary 

In this paper, I have attempted to lay out some issues that have come'out 
of recent research on the change process that have di^'ect implications for pro- 
cess evaluation. These implications are not limited to bilingual education but 
apply equally to other evaluation efforts. However, they clearly apply in bi- 
lingual education, especially given the complexity and lack of clear cut defini- 
tion of what bilingual education is when it is made operational in classrooms, 
buildings and school districts. 



{ 

As previously stated, the change research indicates that evaluators need to^ 
answer two fundamental questions before either formative or summative evaluation 
efforts can proceed. "Is it being used?""and "What is *it'V\ 

In the past, and all too frequently in the present, evaluation studies are 
conducted without having a clear cut answer ta either the use question or the 
configuration question. Consequently, the conclusion "no significant differ- 

ences" may npt be valid, since the evaluation design may have masked differences 

o 

by not separating users of acceptable configurations of the innovation from non- 
users of the innovation or users wha are using unacceptable configurations. 
Such invalid conclusions can lead to unfortunate 'policy decisions. 

This is not meant to say that a", high-f idel ity" perspective has to be taken 
in all implementation efforts. It is quite acceptable in certain situations for 
teachers and other users to be encouraged to create all sorts of adaptations in 
the innovation. In that case the process evaluators need to be able to document 
and describe the .•<iifferent component variations and configurations that result. 
Special staff development activities would have to be developed to address and 
encourage teachers in cj^eating adaptations and diversions. "In either case sum- 
mative evaluation studies must clearly document with first-hand data that the 
innovative group does indeed have* users in it and that any comparison groups are 
comprised of^nonusers. 

Another; implication of the change research is that evaluation must be done 
over time. Change is a process, so longitudinal designs are essential. Evalua- 
tion studies cannot be conducted on just the first year of implementation. 
Formative evaluation data, which are descriptive of the^ implementation process, 
must be constantly returned to program developers, staff developers and others * 
concerned with implementation. A one-time data collection will not provide the 
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required longitudinal perspective that reflects the growth or regression which 
can occur as the change process unfolds. 

In this paper the concept of Levels of Use of the Innovation has been pro- 
posed as a way to answer the first question of "is it being used?" The concept 
of Innovation Configurations has been proposed to address the second question, 
"what, is it?" The concepts and measurement procedures have been briefly de- 
scribed and sample data presented to assist the reader in thinking about these 
concepts and the^r possible imjDl ications f-or their setting. Clearly, formative/ 
process evaluation is a very useful tool. In dealing with a program as compli- 



much clarity as i$ possible about the concepts and variables to be assessed. It 



to program developers, staff developers, users, and policy makers which will 
help them in their task of improving education. 




education, it is essential" to develop as 
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